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ABSTRACT 

A method is described for simulteuieous test 
construction using the Operations Research technique zero-one 
programming. The model for zero-one programming consists of two 
parts. The first contains the objective function that describes the 
aspect to be optimized. The second part contains the constraints 
under which the objective function should be optimized. The selection 
of items is based on information from item response theory. 
Simultaneous test design is used when tests have to be constructed so 
that there is a certain relationship between them. Two examples of 
simultaneous test construction are presented. The construction of two 
parallel tests is considered, and designs of three tests that should 
measure best at successive parts of the ability scale are described. 
Examples were carried out using an item bank of 10 items chosen at 
random. The sequential construction of the same series of tests was 
compared. The examples illustrate that the tests constructed using 
simultaneous techniques besc fit the intentions of the test 
constructor. Three tables give the data for tho test construction 
methods . (Author/SLD) 
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Abstract 



A method for simultaneous test construction, using the 
Operations Research technique zero-one programming Is described. 
The selection of Items Is based on the concept of Information from 
Item response theory. 

Simultaneous test design Is to be used when tests have to *>e 
constructed that have a certain relationship between them. Some 
examples that conclude the paper shdw the advantage of using 
simultaneous tecnnlques over methods that construct tests one after 
another. 
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I. Introduction 



One of the main applications of item banking is t^st 
construction. Recently, a start has been made with research on 
systenatlc test construction based on operations research 
techniques (Boekkool-Tlmninga, 1986; Theunissen, 1985; Theunissen A 
Verstralen, 1986; Tlmmlnga, 1985; van der Linden i Boekkool- 
Tlnmlnga, 1986). The methods developed so far are all methods for 
the construction of one test at a time. 

For many test design problems those methods are applicable; for 
some» however, they will not be appropriate. This Is the case when 
the tests to be constructed have a certain relationship to each 
other, for instance, parallel tests or two-stage testing procedures 
(Lord, 1980). A possible relation is that the tests have no Items 
In common, or that they nave Increasing difficulties or test 
lengths. Of course, such tests can be constructed one at a time, 
but this has some disadvantages. When the first test Is 
constructed, the other tests to be constructed can not be taken 
Into account. A consequence is that an item can be selected for the 
first test while it would fit one of the other tests much bett3r. 
This disadvantage becomes greater the larger the number of tests to 
be designed. 

Following Birnbaum (1968), as an Instrument for test 
construction the concept of Information from Item response theory 
Is used. The Information function for an unbiased consistent 
estimator of ablMty Is defined as the reciprocal of the asymptotic 
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sampling variance of the estimator. For the three-parameter 
logistic model, which considers only dichotomous items, the item 
information function can be expressed as 

aj (1-cJ 

(1) I.(e)- ^ ^ . 

(c^ +exp(a^ ( e-b . ) ) ) ( l+exp( -a^ (e-b^ ) ) r 



where a^, b^, c^ are the discrimination, difficulty and guessing 
parameters for item i and e is the ability level considered. The 
relation between the test 1(e) and item information functions 
(e) is formulated as 

(2) KG) - I I.(G) . 

i«l ^ 

where 1(e) is an upper bound for any scoring method. 

Test construction in this article and the publications mentioned 
above is based on the specification of a target test information 
function. Those items are selected for which the actual test 
information function approximates the desired function best. 
Because it is in general not interesting to consider the whole 
information ^unction, only the values corresponding to some 
specific points at the ability scale are considered. Furthermore, 
it is assumed that the items in the item bank considered are all 
calibrated and fit the same one-dimensional model. 

In this paper a model for simultaneous test construction based 
on zero-one programming is described. Then, the results of two 
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exfinples are given in which two parallel tests and three tests, 
which measure best at successive parts of the ability scale, are 
designed in two different manners: simultaneously and one by one. 
The paper ends with some conclusions and remarks. 

2. A Model for Simultdneous Test Construction 

Thft^ model for simultaneous test construction is based on the 
Operations Research technique zero-one programming. Some references 
to these techniques are: D^ellenbach. George, and McNickle (1983), 
Syslo, ueo, and Ko^jlik (1983), Taha (197S), and Wagner (1972). A 
model for zero-one programming exists of two parts. The first part 
contains th^ objective function which describes the aspect to be 
optimized. The second part of the model contains the constraints 
under which the objective function should be optimized. A 
restriction to both objective functions and constraints Is that 
they all have to be linear. 

In this section first the symbols used In the test construction 
model are given. Next, a number of possible constraints are 
formulated. The section ends with some useful objective functions 
in simultaneous test design. 

2.1. Definitions of Symbols 

Constants 

1 : items in the bank (1 ^ 1,...,N) 
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Ik 



^tk 



total number of items to be selected 

number of Items to be selected for test t 

tests to be constructed ^t - T) 

points on the ability scale to be considered (k « 1,...,K) 

Information of Item 1 at ability level k 

target test information for test t at ability level k 



Variables 



x^^ : decision variable 



1 item i is selected for test t 
^ 0 otherwise 



2.2. Some Constraints 



In test design many constraints may apply. Some of the most 
frequently occuring constraints In simultaneous test design will be 
mentioned here. For a more thorough description of possible 
constraints In test design the reader Is referred to van der Linden 
and Boekkoo1*T1mm1nga (1985). In this reference constraints are 
given for the case that one test at a time is constructed. These 
are however easily generalized for the case of simultaneous test 
design^ by adjusting t.he decision variables. 

The first constraints to be mentioned involve the range of 
values of the decision variables. They may only take the values 
zero and one. Indicating for each test if item i must be part of it 
or not. 
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(3) 



1 « 1 N 

t = 1 T 



The specification of the target Information functions for the 
tests to be constructed requires a constraint for each test at each 
of the ability levels considered. The desired test information for 
test t at ability level k Is given by 



The sign in this formula will be "greater than" in most instances, 
because generally it is preferred getting more information than 
desired than less. 

It is optional to put constraints on the number of items to be 
selected. Only in some Instances the number of items should not be 
fixed. This Is the case when the number of items is part of the 
objective fuction, for instance » when the number of items to be 
selected has to be minimized. For each test the exact number or an 
upper or lower limit on the number of items in the test can be 
specified. For test 1 this is formulated as 



It Is also possible to put restrictions on the total number of 
Items for all tests to be selected 



(4) 



Jj*1t^ilc < ^tk 



N > 



(5) 



1-1 < ^ 
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N T 



« n 



(6) 




It ^ 



Further, It Is possible to put constraints on the ratio of the 
number of items selected for each test, described by a|,a2f*»aj 



Besides putting constraints on the number of items to be 
selected for the tests. It Is also possible to influence the 
Individual selection of items* A simple way to include or exclude 
some specific Items Is to fix the decision variables of these Items 
to one or zero, for Instance, excluding item 2 from all tests 

(8) ^zt"^ ^ ' 1.-. J , 

or Including Item 2 In one of the T tests 
T 



or Including all items in maxima" one test 
T 

(10) X < ^ ^ ' ^•••••^ 
t«i 

In the same manner it is also possible to specify for some of the 
tests that they may have no items in common. 




N N 



N 




(9) 
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2.3. Objective Functions 

In this paragraph some objective functions are given that can be 
used In simultaneous test construction. First, some functions are 
described that are also to be used In the construction of one test 
at the time. Then, a few objective functions are mentioned to be 
used for simultaneous test design only. 

The first objective function that was Introduced In test 
construction by zero-one programming Involved the number of items 
to be selected (Theunlssen, 1985). Theunlssen considered the 
construction of one test at a time as a multi -dimensional knapsack 
problem (e.g., Taha, 1975). He minimized the number of items to be 
selected subject to the constraints that the actual test 
Information at some specified ability levels should exceed certain 
predetermined heights. In simultaneous test construction this 
objective function Is also applicable. The total number of Items to 
be selected for al? tests or one of the tests Is then minimized. 
When the total number of items to be selected for all tests to be 
constructed is minimized, the objective function looks like 

N T 

(11) minimize I I x. 

1-1 t=l 

Objective functions Involving the number of items to be selected 
have both an important advantage and disadvantage. The advantage is 
that with a minimum number of items a maximum amount of information 
can be obtained. The disadvantage concerns the practical 
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applicabili^ of the function; many test constructors may ^ant to 
have the opportunity to Influence the number of Items to be 
selected. 

An objective function that does not have this disadvantage 
Involves minimizing the distance between the target and the actual 
test Information function. This objective function In test 
construction was first looked at by Timminga (1985). It was 
formulated for the construction of one test as minimising the sum 
of the distances between the target and actual test information 
function over all ability levels considered. This is equivalent to 
minimizing the sum of the actual test Information over all ability 
levels, under the constraints that for each ability level 
actual test Information exceeds the value of the target function 
D|^. For each item a decision variable was Introduced indicating 
for item 1 If it "nust be Included In the test or not. This 
objective function Is formulated as follow? 

N K 

(12) minimize V I x.l 

i«l k»l ^ 

subject to 

N 

(13) .^^*i^k ^ \ ^ ' 1.....K 

Another objective function Is to maximize this sum under the 
restriction that the actual test information should not exceed the 
target function value. This function ib not looked at any further 
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here. Formulated for a simultaneous test design nK)del» (12) and 

(13) are transformed into 

N K T 

(14) minlinlze I I I x.A. 

i«l k'l t=l 

subject to 

(15) j/lt^k^^k 

where the actual test Information of all tests Is summed In (14). 
In (15) for each test at each ability level considered a minimum 
height of the test Information to be obtained Is given. 

Objective functions are viewed more Intensively In van der 
Linden and Boekkooi-Tlmmlnga (1986), where the construction of one 
tcit at a time is considered. The objective functions given there 
can be easily generalized to the case that more than one test has 
to be constructed by adjusting the decision variables. The same was 
done for instance by transforming the objective function (12) 
(13) into (14) . (15). 

Besides objective functions that are used in constructing one 
test at ^ time and that can be generalized for the construction of 
more tests at a time, there are also functions to be used 
exclusively in simultaneous test design. These are objective 
functions that Involve the actual test information functions of the 
tests constructed. For Instance, when parallel tests have to be 
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constructed, minimizing the sum of the absolute distances between 
the actual test Information functions. Also, functions like 
mlnlnlzlng the maximum absolute distance between the actual test 
Information functions for the tests constructed are possible. For 
the construction of two tests at the sauie time this Is formulated 
as 

(16) minimize y 
subject to 



N N 

^I^x.^I., - y <^I^x.2l^, k » 1 K 



N N 



where y Is the maximum distance between the two realized test 
Information functions. The ultimate c'loice of an Objective function 
will depend on the specific test design problem and the wishes of 
the test constructor. 

3. Some Examples 

In this section two examples of simultaneous test construction 
are presented. First, the construction of two parallel tests Is 
considered, and second, the design of three tests, which measure 
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best at successive parts of the ability scale Is described. In both 
cases the tests are constructed simultaneously as well as one test 
at a time. 

The examples were carried out using a fictive Item bank 
containing ten items chosen at random. Only ten Items were 
considered, because of the complexity of solving zero-one problems 
with larger numbers of variables and constraints (Theunlssen, 1985; 
Taha» 1975; Van der Linden & Boekkool-Timmlnga, 1986). 



INSERT TABLE 1 ABOUT HERE 



In Table 1 the item parameters and the Item 1nfor.Tiat1on at three 
ability *evels (e « -1, 0, 1) for the Items In the Item bank are 
given. For each item It Is assumed that the guessing parameter Is 
equal to 0. The item difficulties vary from -1.12 to 0.92; the item 
discrimination Indices have values between 0.57 and 1.60. The 
computer program for zero-one programming that was used was based 
on an implementation on a OEC-2060 mainframe of the algorithm of 
Land and Ooig (1960), which Is a branch-and-bound method. 

The first example concerned the design of two parallel tests. 
Tests are considered parallel If they have the s?ivae test 
Information functions (Samejima, 1977). From a practical point of 
view It Htfas desired that the tests should contain an equal number 
of Items. The actual test Information for both tests at the three 
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ability levels considered was desired to be at least 0.50. The 
tests were designed simultaneously twice using different objective 
functions. First, as an objective function the sum of the distances 
between the target and actual test information functions at the 
specified ability levels was considered. The other objective 
function minimized the maximum absolute distance between the actual 
test information functions of both parallel tests. The simultaneous 
test construction model for the first case was 



10 2 3 

(19) minimize I I X x. I. 

1«1 t=l k=l ^ 



subject to 



(20) 



10 

I 

1-1 



*1t^1k * 



t - 1.2 
k = 1.2.3 



(21) 



10 10 
i«l 1-1 



(22) 



«il * ^2 * ^ 



1 = 1,....10 



(23) 



£ {0.1} 



1 = 1 10 

t = 1.2 



In the case of minimizing the maximum absolute distance (19) was 
replaced by (16) and the constraints (17) and (18) had to be added 
to the test construction model mentioned above. When the tests were 
constructed one at a time, the following test design model for the 
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first test was used 

10 3 

(24) minimize I I x.I.. 

1=1 k«l 

subject to 



10 

(25) I > 0.5 k = 1,2.3 

1»1 ^ 



(26) X. e {0.1} i = 1.....10 

The test design model for the second test was the same, except for 
constraints that had to be added to exclude the Items selected for 
the first test, and to determine that the number of Items to be 
selected should be the same as for the first test. 
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The results of this example considering the design of pardllel 
tests are summarized In Table 2. A remarkable difference between 
the two types of test design was noticed. In constructing the tests 
sequentially there was a greav discrepancy between the actual test 
Information functions of both parallel tests constructed which did 
not occur when the tests were designed simultaneously. Furthermore, 
it was seen that most items were selected for an Individual test In 
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sequential test design. Looking at the Items selected It was 
rewrked that in the case of constructing the two tests one after 
another those Items were selected for the first test which had very 
low Item Information values at the ability levels considered. This 
has to be explained from the objective function used minimizing the 
distance between the target and actual test Information function. 
Thus, more Items had to be selected and the actual test Information 
functions differed more from each other than In simultaneously 
designed tests. In comparing the two simultaneous test constructing 
methods it was seen that the differences between the actual test 
Information functions were smaller when the maximum absolute 
distance between both functions was minimized. This was to be 
expected since the actual test information functions were 
Indirectly connected to each other by the target test information 
function in the objective function. The problem of designing 
parallel tests is considered more explicitly in Boekkooi-Timmlnga 
(1986). 

The second example carried out Involved the design of three 
tests which should measure best at successive parts of the ability 
scale. For the tests it was desired that the actual test 
Information should exceed 0.75 at the ability levels e = -I, 0, 1. 
respectively. Further, it was demanded that the tests had no Items 
In common. The objective function used was minimizing the total 
number of items to be selected for the tests to be constructed. The 
model formulation for simultaneously designing these tests was 
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10 3 

(27) minimize I I x.. 

1-1 t-1 

subject to 



10 

(28) I x.,I., > 0.75 

1-1 



10 

(29) I x.-I.- > 0.75 

i-1 



10 

(30) ^I^x^3l^3 > 0.75 



(31) + x^2 *i3 ^ ^ ^ 1 ^0 



(32) x^^ € {0.1} i » 1 10 

t = 1.2.3 



When the tests were constructed one at a time, the test 
construction snodel for the first test which measured best at 
ability level G * -1. was 

10 

(33) minimize I x. 

i»l ' 

subject to 
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(34) 



10 

I x,I,i > 0.75 
1-1 ' 



(35) 



X. e {0.1) 



i = 1 



I • • • » 



10 



The test design models for the second and third test were similar. 
The only difference was that the ability level considered changed, 
and that extra constraints had to be added to exclude the items 
that were already selected for the other test(s). 



The results of the construction of the three tests are described in 
Table 3. It was seen that in the case of sequential test 
construction no solution was found when the third test was to be 
constructed. The test information for the items not yet selected 
was 0.718, which was below the target value of 0.75. The results of 
the simultaneous design, however, showed that it was possible to 
construct all three tests desired. This example showed clearly the 
advantage of simultaneous test design over sequential test design, 
which does not take the other test(s) to be constructed into 
account. Remarkably, for both methods the actual test information 
function of the first test looked a little strange, because the 
amount of test information obtained at the second ability level was 
higher than that at the first level. This can be explained from the 
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properties of the Items In the Item bank considered, and from the 
test construction mode! In which It Is not expHcltely determined 
that the amount of test Information for each test must be highest 
at the ability level considered. 

4* Discussion 

In this paper It Is shown how tests can be constructed 
simultaneously using zero-one programming. Simultaneous test design 
Is to be used when tests have to be constructed that should have a 
certain relationship to each other. From the examples given, In 
which tests were constructed both simultaneously and sequentially. 
It appeared that the tests designed simultaneously corresponded 
most to the desires the test constructor had. Especially In the 
first example, designing the tests one after another. It Is noticed 
that Items are selected for the first test that should fit the 
second test to be constructed better. This problem does not occur 
when the tests are constructed simultaneously. Of course, some of 
the deviations can be partly explained from the small Item bank 
used and the specific objective function that was minimized, but 
these aspects Influence both types of test construction. 
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Table 1 



Item Parameters and Item Information at Three Aolllty Levels 



Item 


b 


a 


I^{-1) 


1^(0) 


1^(1) 


1 


0.48 


0.79 


0.113 


0.151 


0.150 


2 


-0.29 


1.60 


0.471 


0.607 


0.256 


3 


0.54 


1.41 


0.183 


0.432 


0.448 


4 


0.92 


0.98 


0.110 


0.197 


0.240 


5 


0..* 


0.66 


0.090 


0.108 


0.104 


6 


'1.12 


0.78 


0.152 


0.126 


0.082 


7 


-0.50 


0.59 


0.085 


0.085 


0.072 


8 


-0.52 


1.57 


0.536 


0.524 


0.190 


9 


0.12 


1.46 


0.291 


0.529 


0.362 


10 


-0.35 


0.57 


0.079 


0.080 


0.070 
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Table 2 

Items Selected for Two Parallel Tests 



Items Selected I(-l) 1(0) 1(1) 

Simultaneous Construction 

MiiJmlzIng the sum of distances between target and obtained test 
Information functions 

Test 1 6-9-10 0.522 0.735 0.514 

Test 2 4-7-8 0.731 0.806 0.502 

Minimizing the maximum absolute distance between both obtained test 
Information functions 

Test 1 1-2-6-10 0.815 0.964 0.558 

Test 2 4 - 5 - 7 - 8 0.821 0.914 0.606 



Sequential Construction 

Test 1 4-5-6-7-10 0.516 0.596 0.568 

Test 2 3-8 0.719 0.956 0.638 
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Table 3 



Item Selected for Three Tests which Measure Best at 
Successive Parts of the Ability Scale 



Items Selected I(-l) 1(0) 1(1) 



Simultaneous Construction 



Test 1 


8 


- 9 


0.327 


1.053 


0.552 


Test 2 


1 


- 2 


0.584 


0.758 


0.406 


Test 3 


3 


-4-5 


0.383 


0.737 


0.792 



Sequential Construction 

Test 1 2-8 K007 1.131 0.446 

Test 2 3 . ; 0.474 0.961 0.810 

Test 3 Infeaslble 
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